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ABSTRACT 

In the field of educational assessment, many changes 
are going on that are designed to make measurement or assessment 
meaningful to both teachers and learners. Conditions that support 
assessment development and research are identified. The need for 
developing assessments for teachers and learners is being supported 
by research on unintended effects of testing programs, changing 
theoretical underpinnings of assessment, and the independence of test 
development from educational practice. An example of assessment 
development for teachers and learners is the Mathematics Assessment 
Questionnaire (MAQ) , a survey of thoughts and feelings in the 
mathematics classroom for grades 7 through 9. The questionnaire 
provides insight into the meanings that students construct from 
assessment. Technology can help the teacher construct meaning from 
the assessment, as the research of the author and colleagues into the 
use of the MAQ illustrates. Conditions that support assessment 
development and research in New York communities include: (1) new 
organizational arrangements to foster the integration of theory and 
practice; (2) studies that make explicit use of theories of teaching 
and learning; and (3) development of procedures to extend and adapt 
existing assessment development procedures. Eight tables summarize 
points in the discussion. (SLD) 
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Today there are many exciting chsnges going on in the 
field of educational assessment. I broadly characterize 
these changes as the effort to make measur^^ment or 
^ assessment meaningful and useful for teachers and learners. 

That is, there is an effort to define and develop assessment 
£^ FOR teachers and learners, rather than assessment that is 

lO ABOUT teachers and learners. These changes are taking 

^ place in a rapidly shifting context for educational 

^ assessment. This context includes the likelihood that there 

O will be national standards of achievement promulgated for 

Pj| the public education system, and potentially a quasi- 

national or national testing system. 



For educational psychologists, the definition of the 
meaning and use of assessments for teachers and learners has 
a particular importance* For example, in Table 1, the 
definition of meaning requires trying to understand the 
inferences that teachers and learners make from the process 
of assessment (preparation, participation, outcomes) . By 
inferences, we mesai what interpretations or narratives 
teachers and students construct from the assessment process- 
-about themselves as teachers and about themselves as 
learners. By useful assessments, we mean what actions do 
students and teachers take based on assessments. For 
example, teachers may make particular instructional plans 
and students may ask questions of another student or a 
teacher. 

Historically, the field of applied social science 
measurement, including educational assessment, has evolved 
into a profession that is based on general principles* These 
principles are modeled on a scientific paradigm that 
stresses generalizability across settings and thus functions 
more or less independently of the particularities of 
practice* The general principles also assume that 
measurement has an independence of local settings. However, 
assessment for teachers and learners, in the context of 
educational practice, is highly dependent on the local and 
particular, as indicated by the questions in Table 1. 



My concern today is threefold: 



1. to summarize the need for assessment FOR teachers 
^ and learners, specifically, the need for assessment in the 

^ context of practice; 
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2. to briefly provide an example of assessment 
development for teachers and learners bein^ conducted with 
the collaboration of teachers in New York s'hools; 

3, to identify conditions that support assessment 
development and research for teachers and learners, in New 
York communities. 



1. The need for assessment FOR teachers and learners 

Why be concerned about assessment for teachers and learners? 

At least three points indicate the need for concern about 
assessment in the context of practice, ai» identified in 
Table 2: 

Research shows the effects of testing programs on 
practice, effects that are often unintended. And, research 
shows the beliefs about teaching and learning underlying 
most current testing practice do not match the prevalent 
views of educational psychologists. 

Research on tests and testing systems has been sporadic. 

[Early research was primarily concerned with teacher 
attitudes toward and knowledge about test scores (e.g., 
Goslin, 1967; Ward, 1980). With the use of large-scale 
testing systems for accountability and resource allocation, 
empirical research has examined the effects of such systems 
on teacher practice (Ellwein, Glass, & Smith, 1988? 
Haladyna, Nolen, & Haas, 1991; Salmon-Cox, 1981; Smith, 
1991) and the model of teaching and learning underlying the 
test developer's views of education (Shepard, 1991), 
(Information on effects of tests used in evaluation research 
has also been described, cf . Campbell, 1977, on the 
corruption of test data) . ] 

While assessment and measurement is used extensively in the 
work of educational psychologists, it has not been the focus 
of continuing research programs. For example, there is not 
an extensive body of research on the meaning and use of 
external assessments to students and teachers. 



ii. Assessment programs are developing more complex 
performance assessments. There are efforts to integrate 
cognitive theories and assessment theory. 

The changing views of teachers and learners provided by 
cognitive (and social) constructivist theory has already 
influenced assessment development (Tittle, 1990) . One area 
in which the influence is felt in the development of complex 
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performance assessments and in related models of measurement 
(Mislevy, 1989) . For e::ample, there are ass^essment tasks 
that have open-ended formats and may have more than one 
answer or solution possible. There are group problem 
solving situations and writing tasks set in out-of -school 
situations. There are computer simulations and laboratory 
experiements, and there are portfolios of student work. 

As an example, reading assessments may have several 
components and scoring based on models of the reading 
process, including mapping texts for structure and content. 
Students may be asked questions about their familiarity with 
the topic (prior knowledge) , their knowledge about reading 
and reading strategies, and their interest in the reading 
tasks. 

However, there is typically a time lag for the 
integration of theory into teacher development and practice. 
Therefore, teachers may be provided assessment information 
based on theoretical models with which many teachers have 
little familiarity. 

An example is provided by the professional and 
curriculum standards of the National Council of Teachers of 
Mathematics. These standards have a cognitive 
construct ivist view of teaching and learning mathematics, 
and were the basis for changes in the California Assessment 
Program. A case study of the California assessment in 
mathematics (Peterson, 1990) included classroom 
observations. One observer (Cohen, 1990) , reported that 
teachers adopted some practices characteristic of a 
cognitive constructivist view of mathematics teaching and 
learning. However, adoption of practices did not mean an 
adoption of the underlying exjistemoloc / required for 
transforming classroom practice. Thus, in this example, 
there is not a good fit at present between theory, changes 
in assessment, and the context of practice. Is this 
unexpected? No, this situation is not unexpected, since 
assessment is being changed with the purpose of bringing 
change in teaching practice. 



iii. There is an historical independence of assessment 
development practitioners from classroom practitioners. 

Professional practice in test and assessment 
development has evolved to be functionally independent of 
practice in educational classrooms. This is not to say that 
teachers and students are not involved in test and 
assessm.ent development. They are, but primarily to provide 
support for already-developed assessment plans or assessment 
procedures. Teachers provide judgements, sometimes write 
tasks, and students provide responses. Teachers are not 
typically an integral part of the assessment development 



process. The Standards for Educational and PsYcholoqical 
Testing (1985) do not discuss roles for professional 
practitioners whan developing tests and assessments. 



In summary, the need for developing assessments FOR 
teachers and learners is supported by i) research on 
unintended effects of testing programs, ii) the changing 
psychv-^logical and educational theory underlying measurement, 
and ii ; the independence of test development from 
educat .onal practice. I propose that research is needed to 
develop an understanding of assessments in classroom 
settings and to support the intended meaning and effects of 
assessment processes and the resulting information* 

I turn now to an example of assessment development for 
teachers and learners* I give this example because it 
provided an opportunity to explore an unusual area for 
assessment and to explore the meanings and use that students 
and teachers make of an assessment. Further, the example 
provides a way of understanding how the university may be 
able to work in collaboration, working toward colleagueship, 
with teachers. The work described here has been supported 
by the Ford and Aaron Diamond Foundations, and the Graduate 
School of CUNy. 



2, An example of assessment development for teachers and 
learners. 

The assessment project 

Let me provide one example of the meanings that 
students may construct from an assessment tool. The 
assessment tool is the Mathematics Assessment Questionnaire , 
and has its origins in research on women, minorities, and 
mathematics achievement and persistence in course taking. 



The Mathematics Assessment Questionnaire, A survey of 
thoughts and feelings (MAQ) is designed for use in 
mathematics classrooms, grades 7-9 (Hecht & Tittle, 1990; 
Tittle & Hecht, 1990). As shown in Table 3, statements in 
the questionnaire ask the learner to reflect on learning and 
doing mathematical word problems to assess: 

— selected student characteristics (self -regulatory 
skills and affective, motivational and attributional 
beliefs) 

— in the context of classroom activities (During class. 
Working with others, and Homework) 

Table 4 provides several examples of these statements. 



To gain some understanding of the meanings that students 
may construct from an assessment tool, we tried-out one set 
of statements that asked students what they do before, 
during and after solving a non-routine mathematics problem. 
For example- students responded YES, NO, or MAYBE to 
statements such as: 

I looked back at the problem to see if my answer made 
sense, and, I drew a picture to help me understand the 
problem. 

In several classes we asked students to turn over the page 
and answer this question after they had worked a non-routine 
mathematics problem: 

How did thinking about these questions help you think 
about how you do math problems? 

Table 5 provides examples of some of the meanings that 7th 
and 8th grade students wrote down. For example, "Answering 
these questions helps me think about things that I don't do 
on math problems." 

Asking students to write down their thoughts and asking 
students to talk aloud as they consider MAQ statements 
provide examples of beginning to understand the meanings 
that learners construct from assessment information, 

A major concern of the project has been to work directly 
with New York City teachers throughout the development 
process • However, as we worked with teachers on thinking 
about how to make student responses readily accessible, we 
found that there was variability in the types of access to 
this information that teachers felt would be useful. As a 
result, the MAQ was adapted for computer administration and 
a computer-based teacher program was developed. 



Using technology to facilitate teacher construction of 
meaning and use of assessments 



Consider, for example, the teacher's task in 
constructing meaning from assessment results, for a class of 
from 15 to 40 students. For each of the students the 
teacher may consider: the student's instructional and 
personal history that may be relevant to her or his 
performance (responses) and to instructional planning; the 
classroom context in which instructional decisions are 
embedded in the learning and organizational environment of 
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the classroon and school; and an array of assessment 
results, such as that provided with the MAQ (Tittle, 1989) • 

In our current work we — Deborah Hecht, myself, Ralph 
Smallberg (Progranuner) , and graduate students — have 
developed programs so that the survey can be computer- 
administered and the results compiled and examined on a 
teacher program disk (Tittle & Hecht, 1992). There have 
been many interactions and collaborations with New York City 
teachers in the development process. It has been 
particularly interesting to examine the manner in which 
teachers explore the unfamiliar assessment information 
provided by the MAQ, made available in an unfamiliar mode, 
using a computer program. 

The assessment infoirmation is unfamiliar to teachers in 
the following way: there are 14 3 statements intended to 
elicit student beliefs about their self-regulatory thoughts 
and behaviors in each of the three activity settings (during 
class, with others, and homework). There are other 
statements in the areas of confidence, anxiety, interest, 
value, motivation, and attributions. These are areas 
complementary to the mathematics topics and skills typically 
assessed by teachers and by external testing. 

Student responses are available to individual 
mathematics teachers in our project through a microcomputer- 
based program, also typically an unfamiliar way for them to 
view assessment information. Figure 1 provides the 
structure of the teacher program, A small set of teachers 
has volunteered to use the survey with a class and to meet 
with us individually to talk about their students' responses 
to the survey. 

The procedures we have used with teachers are based on 
the think or talk-aloud procedures used in some cognitive 
research (Ericsson & Simon, 1984). In this procedure the 
teacher is asked to talk aloud as he or she explores the 
structure of the program, the structure of the assessment 
survey, the psychological constructs in the survey, and the 
responses of her or his individual students and class. 
(There are also help features and instructional strategies 
in the program that have not yet been systematically 
explored with teachers.) 

We have used the transcriptions of these talk-aloud 
sessions to consider the meaning that teachers construct 
from assessment information. In speculating about the 
framework that might describe how teachers develop or change 
in their use of such an assessment tool and program, we have 
drawn on research from the field of evaluation (Hall & 
Loucks, 1977) and from current research on development and 
change of mathematics teachers (Schifter & Simon, in press; 
Franke, Fennema, Carpenter & Ansell, 1992). 



The evaluation work focused on examining Levels of Use 
(LoU) of an educational innovation — in terms of particular 
strategies. The work with mathematics teachers examined not 
only the teacher's adoption of new classroom techniques but 
alsb the change in "the epistemological perspectives that 
informed teachers' instructional decision making," that is, 
how instruction on a particular topic was thought about, 
planned for, and implemented (Schifter & Simon, in press) . 

We have developed a framework to describe teacher 
change using the MAQ assessment (Hecht & Tittle, 1992) . 
This framework is given in Table 6, and proposes four levels 
of understanding, levels of the teacher's constructing 
meaning from the assessment information, using the teacher 
computer-based program (TMAQ) : 



Table 6. Teacher Levels of Understanding of 
Student Responses 



1. Acquiring procedural skills and conceptual structures 

i. acquiring facility with the computer program 
and the general structure of the assessment 
information 

ii. understanding the psychological constructs that 
comprise the assessment information (definitions) 

2. Contextualizing student responses in the psychological 
domain 

i. accessing other, relevant information about the 
student 

ii. interpreting the student or class response 
conditional upon this other information 

3. Using the contextualized information to select or 
develop specific instructional strategies 

4. Internalizing and transforming the assessment 
information about the psychological domain 

into other instructional settings and practices. 

So far we have identified examples of the first three 
levels, in the five teacher transcripts we have examined. 
Level 4 is speculative. 

Table 7 provides examples of the types of statements 
teachers make as they work with the MAQ to construct meaning 
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and consider possible instructional activities. It is also 
clear from the transcripts that we could identify other 
meanings that teachers construct from the assessment report, 
such as knowledge about the self as teacher (Tittle, 1991). 

While the example of the Mathematics Assessment 
Questionnaire is particular, it provides an illustration of 
social science reseax^ch in urban schools that can support 
assessment for teachers and learners, assessment that is 
linked more directly to classroom teaching and learning 
situations. The example also suggests that this area of 
assessment will necessarily link closely to research on 
teaching and learning in subject matter areas. 



3. Conditions that support assessment development and 
research for teachers and learners in New York communities • 

Based on our interactions with teachers and learners, we 
have suggestions for assessment development and research in 
urban communities. The conditions are briefly stated, given 
the time constraints today. The underlying premise is that 
assessment development and research need to be closely 
linked with classrooms — with teachers and learners, and with 
development and research on teaching and learning in the 
subject areas. 

The conditions likely to support such assessment 
development are identified in Table 8: 



1. New organizational arrangements that foster the 
integration of assessment research and educational practice. 

Educational psychologists have actively been concerned with 
more meaningful, complex theories of teaching and learning, 
and with more complex models of assessment. Organizational 
arrangements are needed to support the engineering of these 
ideas into assessment research conducted in educational 
settings. The term engineering is used deliberately, to 
indicate adaptation of ideas or theories in the setting of 
practice. Research at the Center for Research on Evaluation 
(CRESST) at UCLA provides one example of such an 
orgnization, as perhaps does the middle school mathematics 
(QUASAR) project at the Learning Research and Development 
Center (LRDC) at Pittsburgh. 

Engineering requires new approaches to applied 
research. As Bevan (1991) has argued in a science context, 
organizational arrangements, networks, or other structures 
need to establish what are, in essence, communities-- here, 
in our example, communities of teachers and researchers. 
The Institute for Research on Teaching, established in the 
1970s at Michigan State University by Shulman, and Shulman's 



research project on teacher performance assessment come to 
mind as collaborative examples. For Bevan, what he calls 
the three Cs are paramount — communication, collaboration, 
and colleagueship. 

In educational psychology, as an applied field, this 
translates into educational psychologists working with 
teachers and subject matter specialists, building 
communication, collaboration and colleagueship. Because of 
the nature of our goals and interests, these arrangements 
must involve close work with teachers and learners in 
schools or other educational settings. These arrangements 
will take time and effort to establish, for the three Cs to 
occur: communication, collaboration, colleagueship. 



2. Research and development studies that makti explicit 
use of theories (models) of teaching and learning to guide 
studies on the meaning and use of assessment information. 

These studies should encompass a wide range of 
questions and methods. For example, studies can be designed 
to characterize the teacher's levels of understanding of 
assessments and student responses. Others can describe 
students' understandings of assessment tasks and students' 
explanations of their own results. 

Still other examples bf research can be descriptions 
and analyses of the assessment tasks, the procedures used 
for rating/scoring, the reporting of results, and other 
components of reporting systems, for any and all levels of 
aggregation. These analyses can examine the extent to which 
aspects of the assessment system — tasks, administration 
procedures, scoring, reporting, aggregating, are congruent 
with an intended model of teaching and learning. 



3. Development of a set of procedures to extend and adapt 
the existing set of assessment development procedures, 

(The discussion here is based on work with Garlie 
Forehand (Tittle & Forehand, 1992).) We have described a 
construct ivist perspective on assessment, starting from a 
particular point of view. This point of view recognizes 
that there are several sets of representations that need to 
be understood and examined throughout the process of 
assessment development. Here I will give examples for 
teachers and students, although other concerned participants 
include parents, principals, among others. 

For example, assessment development procedures might 
well include examining the meaning teachers construct from 
an assessment task, to answer questions such as: 



"What existing representations do teachers have about a 
learner's performance in problem solving? How are 
representations modified by particular assessment 
information? How do teachers explain a learner's assessment 
performance? What instructional actions do teachers suggest 
based on assessment? ..." (Tittle & Forehand, 1992^ p. 9). 

Assessment development procedures might examine questions 
about students such as: 

"What meaning do students assign specific messages 
about their performance? What features influence external 
versus internal attribution? What is the relation between 
the instructional function of feedback and the effect on 
self concept? How do students construct meaning from 
holistic analyses of complex performance? How much do 
students understand about the process of judging 
performance? ..." (Tittle & Forehand, 1992, p. 11). 



Current procedures examine student responses to develop 
indices of task characteristics and charactize a learner's 
development. New procedures are needed to describe types of 
task meaning and explanations of performance that are 
reported by representative groups of teachers and students, 
as well as those meanings intended by assessment developers. 
These procedures would provide links to established and 
proposed procedures for validation of complex, performance- 
based assessments (Linn, Baker^ and Dunbar, 1991) . Specific 
criteria for the rating schemes used to evaluate a 
performance task are described by Quellmalz (1991) , and 
Valencia and Calfee (1991) have discussed concerns that need 
specification for portfolios used in assessment. 

The criteria proposed by Linn and his colleagues, and 
others, will extend existing practice in validation 
research. They are concerned with consequences of 
assessment (corruption of the meaning of the assessments due 
to coaching and other unintended consequences) , fairness, 
transfer and generalizability , among other criteria. The 
criteria arise from experiences with existing multiple 
choice tests and preliminary experiences with more complex 
perf ^rmance tasks . 



Summary 

Our experiences on the Mathematics Assessment Project 
suggest that teachers are interested in collaborations, and 
that their collaborations are necessary conditions for 
assessment development in New York communities. It takes 
time and trust to establish the necessary conditions for 
assessment for teachers and learners. 
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Universities have much to offer in teacher 
collaborations* However, to collaborate in development and 
research on assessments that are for teachers and learners, 
we need to build relationships, communications, that 
establish the understanding that teachers, and students, are 
colleagues . 

The promising developments in assessment for teaching 
and learning require renewed efforts to foster 
educationally-relevant meanings and uses of assessments. 
Some of the more complex performance assessments have 
scoring components and procedures based on models of 
cognitive processes and expert performance, as wel] as more 
global educational outcomes. To the already existing 
complexity of the classroom setting, we are adding 
assessment information based on theoretical models with 
which many teachers (and students) have little familiarity. 
Building and supporting collaborative assessment development 
and research networks for teachers and university groups 
will be critical for long-term change to assessments that 
are meaningful and useful for teachers and students. 

I have drawn my examples of research primarily from 
educational classroom settings, viewing teachers as 
professionals. Similar examples could be developed for 
counselors, school psychologists, and subject matter 
supervisors (e.g., mathematics coordinators and 
supervisors) . The diacussion and many of the general 
principles I suggest here for assessment research in the 
context of classroom practice will generalize to other 
professionals using psychological assessments, to clinical 
psychologists and industrial organizational psychologists, 
as well, to foster understandings that will maximize the 
meaningfulness of measurement • 



Paper presented at the Symposium for the inauguration of 
President Frances Degan Horowitz: Children are our future: 
Social science research in New York communities, September 
16, 1992 
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TABLE 1 



Assessment for Teachers and Learners: 
Assessment in the Context of Practice 



Assessments that are meaning ful 

What inferences do teachers and learners make based on 
an assessment? 

For example, what interpretations and narratives do 
teachers construct about their own performance and 
about their students' performances? 

For example, what interpretations do students construct 
about what they know and what is expected of them? 

Assessments that are useful 

What actions do teachers and students take based on 
assessments? 

For example, what instructional plans and decisions do 
teachers make? 

For example, what parts of assessment information do 
students remember and act on? 
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TAB1E2 



Reasons for concern about assessment 
use by teachers and learners 

1. Research shows the effects of testmg programs on 
practice, effects often unintended. And, research 
shows many tests are developed based on beliefs about 
teaching and learning that are not the beliefs in 
curriculum reforms. 



2. Assessment programs are developing more complex 
performance assessments. 

• Changing psychological and educational theory 

• For example, development of assessments based 
on models of the reading process and theory of 
teaching and learning mathematics 

• Time lag between theory and assessment 
development and adaptation to educational 
practice 

3. Historical independence of assessment development 
from classroom practice. 
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TABLE 3 

The Mathematics Assessment Que^tiomiaire: 
A Survey of thoughts and feelings 

Statements ask learners to reflect on learning and doing 
mathematical word problems to assess 

student characteristics 

1. awareness of self-regulatory skills and beliefs 

2. affective, motivational, and attributional beliefs 

in the context of classroom activities 

1. During class, when a teacher leads a lesson 

2. Working with others, in a problem solving group 

3. Doing homework, an independent activity 

) 
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TABLE 4 



Sample Statements from the Mathematics Assessment 
Questionnaire Illiistrating Constructs and 
Classroom Activity Settings 



Self regulation: During class 

When I can think of another way to solve a word problem, 
I volunteer to show the class. 



Anxiety; Working with others 

I dread the thought of trying to solve a math word problem 
with other students. 



Internal learning goals: Homework 

I like to do hard homework math word problems because I 
learn more math by working them. 
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TABLES 

Sample answers to the question: 
How did answering these questions help you to 
think about how you do math problems? 



It gave me more ways to work a problem. 

Answering these questions made me think about how I 
solved the problem and made me double check my 
answer. 



It helps me think about what to do whenever I have 
math problems and it also helps me recall whether I 
check my work or not. 



Answering these questions helps me think about things 
that I don't do on math problems. 



The questions didn^t really help me thmk about how you 
do math problems. But it gets you thinking. 



Carol KdrTiUk 



Figure 1 



STRUCTURE OF THE TEACHER PROGRAM: 
FLOW CHART OF TMAQ 
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TABLE 6 

Teacher Levels of Understanding of Student Responses 

1. Acquiring procedural skills and conceptual structures 

i. acquiring facility with the computer program and 
the general structure of the assessment 
information 

ii. understanding the psychological constructs that 
comprise the assessment information (definitions) 

2. Contextualizing student responses in the psychological 
domain 

i. accessing other, relevant information about the 
student 

ii. interpreting the student or class response 
conditional upon this other information 

3. Using the contextualized information to select or 
develop specific mstnictional strategies 

4. Internalizing and transforming the assessment 
information about the psychological domain into other 
instructional settings and practices. 
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TABLE? 



SAMPLE QUOTES FOR LEVELS 1, 2 AND 3 
Level 1 

Acquiring procedural skills and conceptual structures 

"I like the summary of their responses. Where can I get 
that? I like to see the needs and strengths... I'm looking at 
the summary chart." 

Level 2 

Contextualizing the students' responses in the particular 
psychological domains 

"...Julie ... I wouldn't suspect that she didn't like working 
with other students but now that I think of it she is one who 
likes to stay at her own desk in another part of the room ..." 

Level 3 

Using information from the MAQ to select or develop 
specific instructional strategies 

"Confidence ....she probably doesn't like when I put her in 
a group which is why she's withdrawn and that would either 
make me want to pair her up with somebody that could help 
her with her anxiety about this or make sure that I give her 
individual attention. . . " 



o 
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TABLES 



Conditions necessary to support assessment development and 
research for teachers and learners in New York communities 



• Organizational arrangements that foster the integration of 
assessment R&D and educational practice 



o Communication, collaboration, colleagueship 



• Research and development studies that explicitly use 
theories of teaching and learning 

o Engineering, adaptation of theory and practice 



• Development and adaptation of procedures to understand 
meaning and use of assessments; asking questions such as 

o What existing conceptions do teachers have about a 
learner's problem solving? How are these modified 
by particular assessments? 

o How do students construct meaning from holistic 
analyses of complex performance? How much do 
student's understand about an evaluation process? 
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